Rating aggregation and propagation mechanism for hierarchical services and products

ABSTRACT

Mechanisms are provided, in a hierarchical feedback aggregation (HFA) system implemented in one or more data processing systems, for collecting and presenting user feedback information for a composite offering. A backend engine of the HFA system, implemented in a first data processing system, registers a hierarchical feedback model for the composite offering. A frontend engine of the HFA system, implemented in a second data processing system, receives user feedback for an identified component of the composite offering. The backend engine of the HFA system generates an aggregate user feedback score for the identified component based on a combination of the user feedback for the identified component and aggregate user feedback scores for child components of the identified component in the hierarchical feedback model. The backend engine outputs a representation of the generated aggregate user feedback score for the component to a user.

This application is a continuation of application Ser. No. 14/680,427,filed Apr. 7, 2015, status pending.

BACKGROUND

The present application relates generally to an improved data processingapparatus and method and more specifically to mechanisms for providingrating aggregation and propagation for hierarchical services andproducts.

With the wide availability of products and services through electroniccommerce platforms, with a large number of similar offerings to choosefrom, it has become important to accurately assess and evaluate thequality of the offerings. That is, given two similar services, forexample, users find it important to be provided with information toassist them with deciding which service to select for their use. Inorder to provide such information, may providers of services and/orproducts have implemented a user/customer feedback mechanism by whichthe user/customer can provide information indicative of the user'sperceived quality of the service or product. It has been found thatusers/customers trust such peer feedback much more than any vendor'smarketing or advertisements since they believe the peer feedback helpsthem to make more informed decisions and obtain a best value for theirinvestment of time and/or money.

On the other side, electronic commerce platforms and developers/sellersobtain such peer feedback information to assist them in improving thequality of their offerings. That is, by obtaining feedback from theusers/customers as to where quality is lacking, the providers of theelectronic commerce platforms may take action to modify the electroniccommerce platform to improve the user/customer's experience interactingwith the platform. With regard to developers/sellers ofproducts/services, the developers/sellers may collect such peer feedbackto improve quality of their products/services. Moreover, with thecontinuous move towards cloud services and application stores, peerfeedback has become even more important as the offering providers needquick feedback through the development and operations process to improvetheir offerings at every iteration.

SUMMARY

In one illustrative embodiment, a method, in a hierarchical feedbackaggregation (HFA) system implemented in one or more data processingsystems, each data processing system comprising a processor and amemory, for collecting and presenting user feedback information for acomposite offering. The method comprises registering, by a backendengine of the HFA system implemented in a first data processing system,a hierarchical feedback model for the composite offering. The methodfurther comprises receiving, via a frontend engine of the HFA systemimplemented in a second data processing system, user feedback for anidentified component of the composite offering. Moreover, the methodcomprises generating, by the backend engine of the HFA system, anaggregate user feedback score for the identified component based on acombination of the user feedback for the identified component andaggregate user feedback scores for child components of the identifiedcomponent in the hierarchical feedback model. In addition, the methodcomprises outputting, by the backend engine, a representation of thegenerated aggregate user feedback score for the component to a user.

In other illustrative embodiments, a computer program product comprisinga computer useable or readable medium having a computer readable programis provided. The computer readable program, when executed on a computingdevice, causes the computing device to perform various ones of, andcombinations of, the operations outlined above with regard to the methodillustrative embodiment.

In yet another illustrative embodiment, a system/apparatus is provided.The system/apparatus may comprise one or more processors and a memorycoupled to the one or more processors. The memory may compriseinstructions which, when executed by the one or more processors, causethe one or more processors to perform various ones of, and combinationsof, the operations outlined above with regard to the method illustrativeembodiment.

These and other features and advantages of the present invention will bedescribed in, or will become apparent to those of ordinary skill in theart in view of, the following detailed description of the exampleembodiments of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention, as well as a preferred mode of use and further objectivesand advantages thereof, will best be understood by reference to thefollowing detailed description of illustrative embodiments when read inconjunction with the accompanying drawings, wherein:

FIG. 1 is an example diagram of a distributed data processing system inwhich aspects of the illustrative embodiments may be implemented;

FIG. 2 is an example block diagram of a computing device in whichaspects of the illustrative embodiments may be implemented;

FIG. 3 is an example block diagram of the primary operational elementsof an on-demand hierarchical aggregate feedback system in accordancewith one illustrative embodiment;

FIGS. 4A-4E are example diagrams of hierarchical structures forofferings which may be used to provide on-demand hierarchical aggregatefeedback in accordance with one illustrative embodiment;

FIG. 5 illustrates example diagrams of hierarchical structures fordifferent emergency response services to illustrate the use ofsimilarity of nodes for inferring feedback information for nodes inaccordance with one illustrative embodiment;

FIGS. 6A-6C are example diagrams of a generalized hierarchical structurefor illustrating aggregate feedback generation and propagation inaccordance with one illustrative embodiment; and

FIG. 7 is a flowchart outlining an example operation for providingon-demand hierarchical aggregate feedback in accordance with oneillustrative embodiment.

DETAILED DESCRIPTION

The illustrative embodiments provide mechanisms for peer feedbackaggregation and propagation directed towards hierarchical offerings,e.g., services and/or products. The mechanisms of the illustrativeembodiments recognize peer feedback, e.g., user ratings, as acombination of multiple peer feedback components, or separate userrating components, rather than a single peer feedback or rating directedto quality. For purposes of the present description, peer feedback willbe assumed to be provided in the form of a user ratings which arequantifiable. However, it should be appreciated that the illustrativeembodiments are not limited to such and any peer feedback form can beutilized with the mechanisms of the illustrative embodiments as long asthe peer feedback is able to be aggregated and propagated in the mannerdescribed herein.

In one illustrative embodiment, user ratings (peer feedback) is acomposite of a first rating component and a second rating componentwhere the first rating component is directed to a user or customer'ssubjective evaluation of the quality of the services or products and thesecond rating component is directed to a user or customer's subjectiveevaluation of the relevance of the service or product to the user orcustomer's needs. These ratings are aggregated based on the relevancebeing a weighting applied to the quality rating such that a singlerating is provided for the service or product. Moreover, various otherweightings may be applied during aggregation of ratings as will bedescribed hereafter, e.g., weightings due to edge weights in ahierarchical model, weightings due to user/customer evaluations, and thelike.

The resulting aggregated single rating value for a service or product isthen propagated up a defined hierarchy associated with the service orproduct such that the rating value of the service or product is used toinfluence the rating values (also referred to herein as simply the“rating”) of other entities associated with the service or product. Thedefined hierarchy comprises nodes and edges organized in a hierarchicalmanner, where nodes represent related entities, one of which is theservice or product, and edges represent relationships between theentities. Edges may have associated weights or strengths correspondingto the relevance of the rating of lower nodes to the rating of highernodes in the hierarchy. Weighted calculations are utilized to quantify arating of higher nodes based on a combination of the rating of thehigher node itself and a contribution of the rating of lower nodes.Moreover, nodes that do not have a corresponding rating (i.e. directrating) may have their rating inferred (i.e. an indirect rating iscalculated) from similar nodes elsewhere in the same or differenthierarchy based on a determined level of similarity.

It should be appreciated that the terms “services” and “products” in thepresent description are used to represent entities or items that can bepurchased, used, or viewed by end users/customers. That is, a “service”is any entity or item that provides a function or performs work onbehalf of a user or customer. A “product” is an item that is able to bepurchased or obtained by a user or customer for use by the user orcustomer. In the context of computer based “services” and “products”,the services and products may be provided via dedicated hardware,software executing on specialized or general purpose hardware, or anycombination of hardware and software. Both “services” and “products”will be referred to herein collectively as “offerings” where theofferings are provided by a provider which may be any entity, computersystem, or the like, that makes the offerings available to users orcustomers for use.

As mentioned above, user/customer feedback mechanisms have been known inthe electronic commerce (E-commerce) domain, but these feedbackmechanisms are limited to an atomic electronic commerce offering. Thatis, each electronic commerce feedback mechanisms allows users/customersto provide only quality feedback for a single service/product. Thesefeedback mechanism utilize averaging for the feedback aggregation overmultiple users/customers, however, average feedback does not alwaysreflect the true quality of the offering, especially when there is notenough feedback data available. For example, for the following 3different sets of user/customer feedback, in the form of a 5-star ratingscale, the average is the same but the true quality may not be the same:(4) vs. (4,4) vs. (5,3,4)=>4 vs. 4 vs. 4

Furthermore these feedback mechanisms perform their aggregation, oraveraging, on flat feedback for an atomic or single offering and do notconsider any hierarchical relationships among offerings or otherentities. In addition, these feedback mechanism allow only a single formof feedback to be provided, e.g., a single 5-star rating scale forrating quality. The present invention recognizes that feedback may beprovided in many different forms as different rating factors includingratings according to a defined scale (e.g., a 5-star scale), helpfulnessvotes, natural language comments, or the like. Having multiple types ofrating factors makes aggregation of feedback more complex such thatsimple averaging of a single type of rating, as is done in knownfeedback mechanisms, is not possible.

The illustrative embodiments provide mechanisms for aggregatinguser/customer feedback using multiple types of rating factors and thenpropagating these aggregated feedback values through the hierarchicalrelationships with other entities. The illustrative embodiments may beprovided as an on-demand service that utilizes a frontend component thatmay be deployed to an offering provider's computing system, e.g.,server, and operates to both gather feedback input from users/customersas well as present feedback information to those users/customers fortheir use in making decisions. The illustrative embodiments furtherprovide a backend component that provides the computational aspects ofthe illustrative embodiments with regard to aggregating the variousrating factors provided by the users/customers and propagating them torelated entities in a defined hierarchy.

Thus, the illustrative embodiments offer on-demand feedback collectionand presentation through an embeddable frontend component, such as awidget or the like. The backend component performs automatic feedbackaggregation in a transparent and asynchronous manner without anyinterference with the functionality of the offering (e.g., service) thatuses the feedback information. The backend component provides anindependent Application Programming Interface (API) that is not tied toa particular type of frontend component, such that the backend componentcan be used with a plurality of different frontend components. Thebackend component exploits the hierarchical relationships amongdifferent nodes for feedback propagation and aggregation to have a moreaccurate and fine-grained feedback scoring of the various entities ofthe hierarchical offerings.

The feedback from the users/customers is received as a plurality offactors, such as a quality rating and a relevance rating, which may beprovided in different formats or types. The quality rating is a metricfor the quality of the offering while the relevance is a metric thatquantifies how useful the offering is to satisfy the needs of theuser/customer.

The on-demand feedback mechanisms of the illustrative embodiments may beprovided as a building block and composed with other services to developa broader and richer solution. The on-demand feedback mechanisms of theillustrative embodiments reduce the time to build new solutions whichhave a requirement for user/customer feedback. The ability to quicklystart monitoring offerings for user/customer feedback helps to providerapid feedback and make updates just in time for the next version of theoffering, thereby reducing the delivery cycle time.

The details of example implementations of the illustrative embodimentswill be described hereafter utilizing example equations, procedures, andapparatus for implementing the mechanisms of the illustrativeembodiments. In the description of these example implementations, itwill be assumed that the offering being evaluated is a service offeredby a provider via one or more computing devices via one or more datanetworks. It will further be assumed that the feedback is collected fromusers of the service. The service is being used in this description asan example for ease of understanding with regard to the hierarchicalpropagation of ratings since it is easier for individuals to recognizeservices as having multiple levels of components and related services.However, it should be appreciated that products also have multiplelevels of components and may likewise be the subject of the mechanismsof the illustrative embodiments. Thus, while services and users of suchservices will be used in the following example implementations of theillustrative embodiments, the illustrative embodiments and the presentinvention are not limited to such and may be applied to evaluation ofany offering.

Before beginning the discussion of the various aspects of theillustrative embodiments, it should first be appreciated that throughoutthis description the term “mechanism” will be used to refer to elementsof the present invention that perform various operations, functions, andthe like. A “mechanism,” as the term is used herein, may be animplementation of the functions or aspects of the illustrativeembodiments in the form of an apparatus, a procedure, or a computerprogram product. In the case of a procedure, the procedure isimplemented by one or more devices, apparatus, computers, dataprocessing systems, or the like. In the case of a computer programproduct, the logic represented by computer code or instructions embodiedin or on the computer program product is executed by one or morehardware devices in order to implement the functionality or perform theoperations associated with the specific “mechanism.” Thus, themechanisms described herein may be implemented as specialized hardware,software executing on general purpose hardware, software instructionsstored on a medium such that the instructions are readily executable byspecialized or general purpose hardware, a procedure or method forexecuting the functions, or a combination of any of the above.

The present description and claims may make use of the terms “a”, “atleast one of”, and “one or more of” with regard to particular featuresand elements of the illustrative embodiments. It should be appreciatedthat these terms and phrases are intended to state that there is atleast one of the particular feature or element present in the particularillustrative embodiment, but that more than one can also be present.That is, these terms/phrases are not intended to limit the descriptionor claims to a single feature/element being present or require that aplurality of such features/elements be present. To the contrary, theseterms/phrases only require at least a single feature/element with thepossibility of a plurality of such features/elements being within thescope of the description and claims.

In addition, it should be appreciated that the following descriptionuses a plurality of various examples for various elements of theillustrative embodiments to further illustrate example implementationsof the illustrative embodiments and to aid in the understanding of themechanisms of the illustrative embodiments. These examples intended tobe non-limiting and are not exhaustive of the various possibilities forimplementing the mechanisms of the illustrative embodiments. It will beapparent to those of ordinary skill in the art in view of the presentdescription that there are many other alternative implementations forthese various elements that may be utilized in addition to, or inreplacement of, the examples provided herein without departing from thespirit and scope of the present invention.

It should further be understood that the present invention may be asystem, a method, and/or a computer program product. The computerprogram product may include a computer readable storage medium (ormedia) having computer readable program instructions thereon for causinga processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Thus, the illustrative embodiments may be utilized in many differenttypes of data processing environments. In order to provide a context forthe description of the specific elements and functionality of theillustrative embodiments, FIGS. 1 and 2 are provided hereafter asexample environments in which aspects of the illustrative embodimentsmay be implemented. It should be appreciated that FIGS. 1 and 2 are onlyexamples and are not intended to assert or imply any limitation withregard to the environments in which aspects or embodiments of thepresent invention may be implemented. Many modifications to the depictedenvironments may be made without departing from the spirit and scope ofthe present invention.

FIG. 1 depicts a pictorial representation of an example distributed dataprocessing system in which aspects of the illustrative embodiments maybe implemented. Distributed data processing system 100 may include anetwork of computers in which aspects of the illustrative embodimentsmay be implemented. The distributed data processing system 100 containsat least one network 102, which is the medium used to providecommunication links between various devices and computers connectedtogether within distributed data processing system 100. The network 102may include connections, such as wire, wireless communication links, orfiber optic cables.

In the depicted example, server 104 and server 106 are connected tonetwork 102 along with storage unit 108. In addition, clients 110, 112,and 114 are also connected to network 102. These clients 110, 112, and114 may be, for example, personal computers, network computers, or thelike. In the depicted example, server 104 provides data, such as bootfiles, operating system images, and applications to the clients 110,112, and 114. Clients 110, 112, and 114 are clients to server 104 in thedepicted example. Distributed data processing system 100 may includeadditional servers, clients, and other devices not shown.

In the depicted example, distributed data processing system 100 is theInternet with network 102 representing a worldwide collection ofnetworks and gateways that use the Transmission ControlProtocol/Internet Protocol (TCP/IP) suite of protocols to communicatewith one another. At the heart of the Internet is a backbone ofhigh-speed data communication lines between major nodes or hostcomputers, consisting of thousands of commercial, governmental,educational and other computer systems that route data and messages. Ofcourse, the distributed data processing system 100 may also beimplemented to include a number of different types of networks, such asfor example, an intranet, a local area network (LAN), a wide areanetwork (WAN), or the like. As stated above, FIG. 1 is intended as anexample, not as an architectural limitation for different embodiments ofthe present invention, and therefore, the particular elements shown inFIG. 1 should not be considered limiting with regard to the environmentsin which the illustrative embodiments of the present invention may beimplemented.

FIG. 2 is a block diagram of an example data processing system in whichaspects of the illustrative embodiments may be implemented. Dataprocessing system 200 is an example of a computer, such as client 110 inFIG. 1, in which computer usable code or instructions implementing theprocesses for illustrative embodiments of the present invention may belocated.

In the depicted example, data processing system 200 employs a hubarchitecture including north bridge and memory controller hub (NB/MCH)202 and south bridge and input/output (I/O) controller hub (SB/ICH) 204.Processing unit 206, main memory 208, and graphics processor 210 areconnected to NB/MCH 202. Graphics processor 210 may be connected toNB/MCH 202 through an accelerated graphics port (AGP).

In the depicted example, local area network (LAN) adapter 212 connectsto SB/ICH 204. Audio adapter 216, keyboard and mouse adapter 220, modem222, read only memory (ROM) 224, hard disk drive (HDD) 226, CD-ROM drive230, universal serial bus (USB) ports and other communication ports 232,and PCI/PCIe devices 234 connect to SB/ICH 204 through bus 238 and bus240. PCI/PCIe devices may include, for example, Ethernet adapters,add-in cards, and PC cards for notebook computers. PCI uses a card buscontroller, while PCIe does not. ROM 224 may be, for example, a flashbasic input/output system (BIOS).

HDD 226 and CD-ROM drive 230 connect to SB/ICH 204 through bus 240. HDD226 and CD-ROM drive 230 may use, for example, an integrated driveelectronics (IDE) or serial advanced technology attachment (SATA)interface. Super I/O (SIO) device 236 may be connected to SB/ICH 204.

An operating system runs on processing unit 206. The operating systemcoordinates and provides control of various components within the dataprocessing system 200 in FIG. 2. As a client, the operating system maybe a commercially available operating system such as Microsoft® Windows7®. An object-oriented programming system, such as the Java™ programmingsystem, may run in conjunction with the operating system and providescalls to the operating system from Java™ programs or applicationsexecuting on data processing system 200.

As a server, data processing system 200 may be, for example, an IBMeServer™ System P® computer system, Power™ processor based computersystem, or the like, running the Advanced Interactive Executive (AIX®)operating system or the LINUX® operating system. Data processing system200 may be a symmetric multiprocessor (SMP) system including a pluralityof processors in processing unit 206. Alternatively, a single processorsystem may be employed.

Instructions for the operating system, the object-oriented programmingsystem, and applications or programs are located on storage devices,such as HDD 226, and may be loaded into main memory 208 for execution byprocessing unit 206. The processes for illustrative embodiments of thepresent invention may be performed by processing unit 206 using computerusable program code, which may be located in a memory such as, forexample, main memory 208, ROM 224, or in one or more peripheral devices226 and 230, for example.

A bus system, such as bus 238 or bus 240 as shown in FIG. 2, may becomprised of one or more buses. Of course, the bus system may beimplemented using any type of communication fabric or architecture thatprovides for a transfer of data between different components or devicesattached to the fabric or architecture. A communication unit, such asmodem 222 or network adapter 212 of FIG. 2, may include one or moredevices used to transmit and receive data. A memory may be, for example,main memory 208, ROM 224, or a cache such as found in NB/MCH 202 in FIG.2.

Those of ordinary skill in the art will appreciate that the hardware inFIGS. 1 and 2 may vary depending on the implementation. Other internalhardware or peripheral devices, such as flash memory, equivalentnon-volatile memory, or optical disk drives and the like, may be used inaddition to or in place of the hardware depicted in FIGS. 1 and 2. Also,the processes of the illustrative embodiments may be applied to amultiprocessor data processing system, other than the SMP systemmentioned previously, without departing from the spirit and scope of thepresent invention.

Moreover, the data processing system 200 may take the form of any of anumber of different data processing systems including client computingdevices, server computing devices, a tablet computer, laptop computer,telephone or other communication device, a personal digital assistant(PDA), or the like. In some illustrative examples, data processingsystem 200 may be a portable computing device that is configured withflash memory to provide non-volatile memory for storing operating systemfiles and/or user-generated data, for example. Essentially, dataprocessing system 200 may be any known or later developed dataprocessing system without architectural limitation.

With reference again to FIG. 1, one or more of the server computingdevices 104, 106 provides one or more services to users of clientdevices 110, 112, and/or 114. These services may comprise a hierarchy ofcomponents or services and sub-services that are related to one anotherand which together constitute an overall composite service. For example,an E-commerce ordering service may have constituent services such as anorder acceptance service, a seller service, and a payment service whichtogether constitute the overall composite E-commerce ordering service.Likewise, each of these constituent services may have sub-services aswell. Multiple levels or layers of the hierarchical representation ofthe composite service may be present and represented in a hierarchicalmodel data structure of the composite service. Through a registrationprocess, a service provider may register with the on-demand hierarchicalfeedback aggregation system of the illustrative embodiments and providea hierarchical user feedback model of the composite service in whichnodes represent the components or constituent services of the overallcomposite service with edges representing the relationships betweenthese components or constituent services. In short, the nodes representany part of the composite service that may be experienced by an end userand which may be rated by an end user through a feedback mechanism whilethe edges represent the dependencies between components or constituentservices of the composite service.

The server computing devices 104, 106 may host or execute softwareengines or modules that utilized user feedback to perform analytics,generate reports, or provide other output to a provider of the servicesto assist in improving the provider's offerings to users. These softwareengines or modules are referred to herein as service specific feedbackconsoles. These services specific feedback consoles perform additionalanalytical operations and/or output operations specifically desired bythe provider of the services based on feedback analytic results providedby the mechanisms of the illustrative embodiments. Alternatively, theservice specific feedback consoles may provide an interface throughwhich the results of the operations of the illustrative embodiments maybe presented to an administrator or other authorized individualassociated with the service provider for use by the authorizedindividual in making user feedback based decisions regarding theservices provided by the provider.

The mechanism of the illustrative embodiments may interface with theseservices specific feedback consoles via a frontend engine of theon-demand hierarchical aggregate feedback system of the illustrativeembodiments. This frontend engine may be utilized to collect userfeedback information and provide this user feedback information to abackend feedback aggregation and propagation engine that operates toaggregate user feedback of various feedback factor types and propagatethis feedback in accordance with the established composite servicehierarchical user feedback model for the particular composite serviceand its related components or constituent services.

The frontend engine may be deployed to the service specific feedbackconsoles in an on-demand manner. By “on-demand” what is meant is thatthe service provider may request the assistance of the on-demandhierarchical aggregate feedback system when needed and the frontendengine may be deployed into the software environment of the servercomputing device so as to execute and gather feedback information fromusers. Alternatively, the frontend engine may already be deployed intothe server computing device's software environment, such as in responseto a registration process performed by the provider, but may be enabledin an on-demand manner by an on-demand request from the provider, suchas via an administrator console or the like, which submits the requestto the on-demand hierarchical feedback aggregation system of theillustrative embodiments.

The frontend engine operates in a parallel manner with the servicesoffered by the server computing device such that the frontend enginedoes not interfere with the operation of the services and does notrequire a modification of the services to permit the use of the frontendengine. The frontend engine gathers the user feedback as describedhereafter and also provides functionality for providing an output ofpreviously aggregated user feedback to the users and also provides aninterface through which the service provider may obtain user feedbackinformation generated by the backend user feedback aggregation andhierarchical propagation engine.

Another server, such as server 106, may provide the on-demandhierarchical feedback aggregation system of the illustrative embodimentsand service requests from other service providers for on-demand accessto the functionality of the on-demand hierarchical feedback aggregationsystem. As such, the server 106 may provide the frontend engines of theon-demand hierarchical feedback aggregation system to a serviceprovider's server 104 via the network 102 for installation on theservice provider's server 104, for use in gathering user feedbackinformation. Initially, the service provider may follow a registrationprocess with the on-demand hierarchical feedback aggregation system(hereafter referred to as the hierarchical feedback aggregation (HFA)system for simplicity) on server 106 to register a composite servicewith the HFA system and provide a hierarchical feedback model for thecomposite service. The HFA system on server 106 may provide aninteractive tool through which the provider is able to graphicallyconstruct the hierarchical feedback model using nodes and edges torepresent the components of the composite service and theirrelationships (or dependencies).

It should be noted that in such a hierarchical feedback model, the nodesrepresent the components of the composite service for which userfeedback is sought and the edges represent the relationships betweenthese components. These components may represent atomic or compositeservices within the overall composite service, data structures utilizedby the overall composite service, individual persons or components of aservice with which a user may interact (e.g., customer servicepersonnel), or any other entity that a user may interact with whenutilizing the overall composite service or a sub-portion of the overallcomposite service. Edges within the hierarchical feedback model may beassigned weights which represent the amount of dependency of userfeedback on the user feedback of the related nodes. For example, if nodeB is a child node of node A in the hierarchical feedback model, and theedge between node B and node A has a weight of 0.35, then 35% of theuser feedback score generated for node B influences the user feedbackscore for node A, as will be described in greater detail hereafter. Theweight of an edge may be initially set by the provider of the compositeservice during the registration process but may be dynamically updatedbased on evaluations of actual utilization of components of thecomposite service such that the weights may be dynamically adjusted inthe hierarchical feedback model.

Having obtained the hierarchical feedback model for the compositeservice offered by the service provider via their server 104, the HFAsystem may receive a request from the service provider to initiate userfeedback collection and aggregation based on the hierarchical feedbackmodel. In response, if the frontend engine of the HFA system has notalready been deployed to the server 104 in response to the registrationprocess, then the frontend engine is provided to the server 104 anddeployed within the software environment of the server 104 forexecution. This deployment may involve installation of the frontendengine in association with a service specific feedback console in someinstances which provides the user feedback questions for the variouscomponents of the composite service with the frontend engine collectingthe user input to these questions and providing the collected feedbackto the backend feedback aggregation and propagation engine of the HFAsystem on server 106. The questions that are posed by the servicespecific feedback console may be specific to the particular service orcomponent with which the service specific feedback console isassociated, e.g., “How would you rate the quality of the EmergencyResponse Weather Service application?” rather than a generic “How wouldyou rate your experience today?”

Alternatively, the frontend engine may be configured to provide thequestions for presentation to users to solicit feedback and may collectthe resulting feedback and provide it to the HFA system backend engine.In such a case, the frontend engine may comprise a generic questionstructure which is then customized to the specific hierarchical feedbackmodel associated with the composite service with which it is deployed.For example, the set of questions may comprise generic templates thathave fields that are populated for the various nodes of the hierarchicalfeedback model for which feedback is being sought. Thus, the same orsimilar questions may be repeated for various nodes of the hierarchicalfeedback model such that user feedback may be solicited for multiplelevels of the hierarchical feedback model, e.g., “On a scale of 1 to 5,how would you rate the quality of your experience with the ACME PAYMENTSERVICE today?”, “On a scale of 1 to 5, how would you rate the qualityof your experience with the ACME ORDER ACCEPTANCE SERVICE today?”, orthe like, which the capitalized terms are names of nodes in thehierarchical feedback model.

Via the frontend engine, and/or a service specific feedback console ifone is provided, users submit feedback regarding the composite service,and the individual components of the composite service, which iscollected for further aggregation, propagation, and analysis asdescribed hereafter. In addition, the frontend engine may provide aninterface through which users are able to see feedback for differentofferings, e.g., different services provided by the service provider.Thus, in addition to collecting user feedback and submitting the userfeedback to the backend engine of the HFA system, the frontend enginefurther allows users to submit queries or requests for aggregatedfeedback information from the backend engine and provides mechanisms forpresenting or outputting this requested aggregated feedback informationto the user. Thus, the frontend engine has two primary modes ofoperation: an input mode and an output mode. In the input mode, thefrontend engine collects user feedback information. In the output mode,the frontend engine presents the aggregated feedback information tousers via one or more output graphical user interfaces.

As noted above, the feedback information that is received, aggregated,propagated, analyzed, and ultimately provided as an output comprisesmultiple types of rating information. In one illustrative embodiment,the feedback information comprises a rating, which represents thequality of the service as perceived by the user, and a relevance, whichrepresents the importance of the service to the user's needs asperceived by the user. These feedback information factors may beaggregated together to generate a single feedback score for the serviceby utilizing the relevance as a weighting factor to the rating factor,i.e. feedback score=rating*relevance. Each factor may be provided usingan acceptable scale for the particular implementation. The scales foreach factor may be the same or different with a correlation functionbeing used to correlate one factor to another when different scales areutilized. For ease of explanation, it will be assumed that each factoris evaluated using a same type of scale, e.g., a 1 to 5 star scale. Ofcourse, other factors may also be utilized, and other calculations maybe used to generate a single feedback score without departing from thespirit and scope of the illustrative embodiments.

In one illustrative embodiment, in addition to considering the ratingand the relevance, an evaluation of the user providing the feedbackinformation may also be taken into consideration. For example, when auser utilizes the service provided by the service provider via server104, the user may be required to provide some information eitherknowingly or unknowingly. Knowingly, the user may register with theservice provider and provide certain demographic information includingage, gender, ethnicity, geographical home location, occupation, likes,dislikes, etc. Unknowingly, the user's accessing of the server 104 mayallow the server 104 to collect IP address information which can becorrelated to a geographical region, for example, the user's history ofpreviously provided feedback may be compiled and accessed to identifythe user's credibility based on the previously left user feedback, orthe like. Information gathered about the user may be compared againstcertain service provider criteria for evaluating users that providefeedback information. For example, the frontend engine may be able toaccess service provider registered user information to provide to thebackend engine of the HFA system along with the user feedbackinformation so that the backend engine can compare the demographics ofthe users with demographics of interest as registered by the serviceprovider during the registration process and associated with thehierarchical feedback model.

For example, a service provider may have as their primary targetedconsumer, males aged 30-40. If a user that provides feedback informationvia the frontend engine also has demographics indicating that the useris a male aged 35, then the user weight applied to the feedbackinformation may be given a relatively higher user weighting than if theuser were a female aged 55 or a male aged 60, for example. While theseother users may have their user weight relatively lower than the maleaged 35, this does not mean that their weight is necessarily zero, butsome value less than that of the targeted user. Similar evaluations maybe made with regard to user information that is unknowingly provided bythe user, e.g., if the targeted users are users in the New York Citygeographical area and the user's IP address indicates the user to belocated in a region in South Carolina, then the user weight associatedwith the user will be relatively lower than users that actually residein the New York City geographical area. The user weights may be appliedto the aggregate of the rating and relevance discussed above so as toweight the user's feedback information relative to other user's feedbackinformation.

The feedback information collected by the frontend engine is providedback to the backend feedback aggregation and propagation engine (alsoreferred to herein as the “background engine”) for calculation ofaggregate feedback scores, propagating the aggregate feedback scores,and potentially performing other analytical operations on the aggregatedand propagated feedback scores so as to provide hierarchical feedbackinformation back to the service provider in accordance with serviceprovide requests for such hierarchical feedback information. Moreover,the background engine may further dynamically update the feedbackhierarchy structure based on actual utilization of the components of thecomposite service so as to adjust edge weights or the like to moreaccurately represent the dependencies of user feedback between thevarious components of the composite service. In general, the backgroundengine calculates an individual feedback score for a particularcombination of user and node of the hierarchical feedback model andthen, using the hierarchical feedback model and propagation rules andequations established for propagating feedback through the hierarchicalfeedback model, propagates the user feedback for the node to otherrelated nodes in the hierarchical feedback model to thereby update theuser feedback scores for these other related nodes. The details of theoperation of example embodiments of the backend engine will be providedhereafter.

FIG. 3 is an example block diagram of the primary operational elementsof an on-demand hierarchical feedback aggregation (HFA) system inaccordance with one illustrative embodiment. The various elements shownin FIG. 3 may be implemented as computer hardware elements, softwareloaded into, and executing on, computer hardware, e.g., processors andmemories, or any combination of hardware and software elements.Moreover, while the depiction in FIG. 3 illustrates the HFA systemcomprising a single front end engine 326 deployed in a single serviceprovider system 330 with a single backend feedback aggregation andpropagation engine 310, the illustrative embodiments are not limited tosuch. It should be appreciated multiple front end engines 326 may bedeployed into various software environments provided on the same ordifferent computing devices and may communicate with the backendfeedback aggregation and propagation engine (hereafter referred to asthe “backend engine”) 310 via one or more data networks and theapplication programming interface 316 and feedback listener 318 of thebackend engine 310.

As shown in FIG. 3, the HFA system is generally comprised of one or morefront end engines 326 deployed in one or more software environments ofone or more data processing or computer systems, and a backend engine310 the comprises the logic for performing the various backendcalculations and operations for aggregating user feedback informationand propagating this user feedback information through a definedcomposite service feedback hierarchy structure for a composite service.It should be appreciated that the term “logic” as used herein refers toeither hardware of software logic, where hardware logic is typicallyprovided as hardware circuitry elements, e.g., logic gates, wires,buffers, and other types of circuit elements, and software logic isprovided as software instructions and data that are loaded into memoryand executed by one or more processors of one or more data processingsystems or computers.

The backend engine 310 comprises an on-demand registration/requestengine 312 which provides logic for registering composite services foruser feedback information collection, aggregation, and hierarchicalpropagation. The backend engine 310 further comprises logic for handlingon-demand requests from service providers for initiating user feedbackinformation collection, aggregation, and propagation for registeredcomposite services. In operation, when a service provider wishes toutilize the on-demand HFA system of the illustrative embodiments, anadministrator or other authorized individual of the service provider mayutilize an administrator console 332 of the service provider system 330and log onto 302 the HFA system to interact with the on-demandregistration/request engine 312 to register the composite service thatis to be monitored for user feedback information collection,aggregation, and propagation. As part of the registration process, theauthorized user may utilize the on-demand registration/request engine312 to construct, or otherwise submit, a hierarchical feedback model fora composite service that is to be monitored for user feedbackinformation collection, aggregation, and propagation. The hierarchicalfeedback model is stored in the service feedback hierarchy structuresdatabase 314. The hierarchical feedback model comprises nodes and edgesconnecting nodes, where the nodes represent components of the compositeservice, e.g., sub-services, data structures, human resources (servicepersonnel), and other components that together constitute the compositeservice. The edges may be assigned initial weights indicative of therelative dependence of a higher level node on its child lower levelnodes.

In addition to the hierarchical feedback model, the authorized user mayfurther indicate particular users of interest for which user feedbackinformation is most desirable. For example, the authorized user mayspecify the demographic characteristics of users that are their targetfor user feedback information. This target user information may be usedto compare to information about users that leave user feedbackinformation to determine how much or little to weight that particularuser's feedback information, as described hereafter.

The on-demand registration/request engine 312 further comprises logicfor handling on-demand requests 302 for initiating the collection,aggregation, and propagation of user feedback information for aregistered composite service. In response to a request from anadministrator console 332 to initiate user feedback collection,aggregation, and propagation, if the frontend engine 326 has not alreadybeen deployed to a software environment for the composite service in theservice provider system 330, then the frontend engine 326 is deployed tothe service provider system 330. In some illustrative embodiments, thefrontend engine 326 may be provided as an applet, plug-in component, orother software element that does not need a formal installation processand instead is provided as an extension of the software environment. Inother illustrative embodiments, the frontend engine 326 may be installedon the service provider system 330 and may operate as a backgroundapplication that collects user feedback information and sends it to thebackend engine 310.

As shown in FIG. 3, in some illustrative embodiments, the frontendengine 326 may operate in conjunction with a service specific feedbackconsole 336 which provides the logic for presenting questions to a userto solicit feedback information which is then collected and forwarded bythe frontend engine 326. In such a case, the frontend engine 326 may begeneric in nature such that the same frontend engine 326 may be deployedas separate instances in multiple software environments of compositeservices on the same or different service provider systems 330. In otherillustrative embodiments, the frontend engine 326 may be customized tothe specific composite service and may perform the operations of boththe service specific feedback console 336 and the frontend engine 326.In such an embodiment, the frontend engine 326 may have templates forquestions that can be presented to the user for soliciting userfeedback, with fields of the templates being populated by informationregarding the nodes and edges of the composite service's hierarchicalfeedback model stored in the service feedback hierarchy structures 314.

Once the frontend engine 326 is deployed into the software environmentof the composite service of the service provider system 330, and itsoperation is initiated in response to an on-demand request from theadministrator console 332, monitoring of the user feedback informationis performed. The service provider system 330 provides a graphical userinterface 334 through which users may access the services 338 providedby the service provider system 330. The services 338 together mayconstitute a composite service as a whole.

In response to a user selecting to terminate a session with the serviceprovider system 330 or otherwise indicating that the user has finishedutilizing one or more of the services 338, the service feedback console336 may output to the user, via the graphical user interface 334, one ormore questions to solicit feedback from the user. The questions mayutilize a defined rating scale for rating the services 338, or thecomposite service as a whole, with regard to quality and for evaluatingthe services 338 or composite service with regard to relevance to theuser's needs. The rating scale may be different from quality rating thanfrom relevance rating. For example, the quality rating may utilize a5-star rating scale where a higher number of stars selected by the userindicates a higher user perceived quality of the services 338. Therelevance rating may be provided as a positive or negative vote, e.g.,thumbs up or thumbs down indication. Alternatively, all ratings mayutilize the same rating scale.

The frontend engine 326 may further collect information about the userproviding the feedback information. This information may be knowingly orunknowingly (by the user) obtained by the frontend engine 326. Forexample, the user may have an established profile with the serviceprovider system 330 and this profile may be provided along with thefeedback information by the frontend engine 326 to the backend engine310. The user information may also be solicited from the user as one ormore questions presented along with the user feedback informationquestions, e.g., questions asking for the user's age range, gender,economic status, and the like. The user information may be unknowinglyobtained by utilizing background information collected from the headeror metadata of the user's data communications with the service providersystem 330. This information may include, for example, IP address,geographical region information, or any other information that may bepresent in the header or metadata of data communications.

The user feedback information and user information may be collected bythe frontend engine 326 and provided 304 to the backend engine 310 viathe feedback listener 318 which listens for feedback informationtransmissions from one of more frontend engines 326. The feedbacklistener 318 receives the user feedback information and provides it tothe aggregation engine 322. The aggregation engine 322 implementsaggregation algorithms for aggregating and propagating the user feedbackinformation utilizing the composite service feedback model registeredfor the composite service in the service feedback hierarchy structures314. In general, the aggregation engine 322 generates an aggregatefeedback score for the node for which the user feedback corresponds bycalculating a value for a function of the user feedback, relevancerating, weighting due to the user being or not being a target user forthe composite service, and further taking into considerationcontributions from feedback scores of lower level child components ofthe composite service. The aggregate feedback score for the node is thenpropagated up the hierarchy of the hierarchical feedback model of thecomposite service in accordance with the weights of edges in thehierarchical feedback model so as to provide contributions to theaggregate feedback scores of the nodes which are parents of the currentnode.

The resulting aggregate feedback scores for the various nodes of thehierarchical feedback model are stored in the historical data structure324 for future analytical operations and providing of information tousers and/or administrators in performing decisions. For example, thehistorical data 324 may be used to present 306 ratings of components ofthe composite service to a user via the frontend engine 326.Alternatively, the historical data 324 may be accessed 308 by anadministrator or other authorized individual of the service providersystem 330 via the administrator console 332 or anadministrator/authorized individual of the HFA system via theadministrator console 320. The historical data provided to a user maycomprise the aggregate user feedback for a service or component ofinterest. The historical data provided to an administrator may be morein-depth and indicate information about aggregate user feedback as wellas the underlying hierarchical feedback model, for example.

In addition, the user feedback received may be used to update the edgeweights of corresponding edges in the hierarchical feedback model of thecomposite service. For example, the usage of a service or component ofthe composite service by the user, as indicated by the user providingfeedback for the service or component, may be used to update usageparameters associated with the corresponding nodes which may in turncause an update to the edge weights based on the usage. That is, nodesthat are determined to be more utilized are more influential in the userfeedback ratings of other related nodes in the composite service. Thisinfluence fades the further away from the node the other nodes are inthe hierarchical feedback model. Thus, the weights of edges may bemodified by a decreasing amount as one traverses the hierarchicalfeedback model up the edges connecting nodes to the current node forwhich the usage is determined to have changed. Thus, for example, usinga function based on the amount of increase in usage and the distance ofthe edge from the current node for which the usage changed, the edgeweight of an edge may be dynamically updated.

As mentioned above, the illustrative embodiments provide for theaggregation of user feedback and the propagation of the user feedback inaccordance with established composite service hierarchical feedbackmodels. FIGS. 4A-4E are example diagrams of hierarchical feedback modelsfor offerings which may be used to provide on-demand hierarchicalfeedback aggregation in accordance with one illustrative embodiment. Asshown in FIGS. 4A-4E, these hierarchical feedback models may be fordifferent types of offerings including services and products. FIGS.4A-4E illustrate examples of the types of hierarchical feedback modelsthat may be registered with the backend engine 310 of the HFA system.

FIG. 4A is an example diagram of a hierarchical feedback model for acloud service 410. As shown in FIG. 4A, the cloud service comprises asocial intelligence analytics service 411 and a healthcare payeranalytics service 412. The social intelligence analytics service 411instantiates or has versions for a social intelligence analytics (SIA)service for bank 1 413 and an SIA for bank 2 414. Each instance presentsresources to the users which the users may rate separately such that oneinstance of the SIA service may not receive the same user feedback orratings as another. The SIA for bank 1 413 is a parent node of reportnodes 415 and 416, while report node 417 is a child node of SIA for bank2 414. As shown by the arrows in FIG. 4A, which represent edges betweenthe nodes, aggregate feedback for child nodes provides a contribution,and thus influences, the aggregate feedback of the child nodes' parentnodes. Thus, aggregated user feedback for SIA for bank 1 413 influencesthe aggregate user feedback of social intelligence analytics service411, along with aggregate user feedback for SIA for bank 2 414.Likewise, the aggregate user feedback for social intelligence analyticsservice 411 influences the aggregate user feedback for the cloud service410, along with the healthcare payer analytics service 412.

FIG. 4B illustrates a similar hierarchical feedback model that may beestablished and registered for a movies 420. At a first level, differentmovie genres are identified as nodes 421 and 422. At the next level,movie series are identified, including the movie series for theGodfather movies 423. At the next lower level, there are individualnodes representing the individual Godfather movies in the Godfatherseries, i.e. Godfather 424, Godfather II 425, and Godfather III 426. Ina still lower level, individual cast members for each of the movies maybe identified as separate nodes, such as Marlon Brando 427 and Al Pacino428.

It should be appreciated that a user may provide user feedback for anyof the nodes in the hierarchical feedback model. However, in knownfeedback mechanisms, there is no consideration of the dependencyrelationships among the nodes in the hierarchy, i.e. a flat userfeedback algorithm is utilized. As discussed above, the illustrativeembodiments propagate this user feedback up the hierarchy in accordancewith the weights of edges between nodes linking to the node for whichuser feedback was provided. In FIGS. 4A-4E, the weights of the edgeshave not been shown for simplicity but the use of such weights withregard to propagating aggregated user feedback will be described ingreater detail hereafter with regard to FIGS. 6A-6C.

FIG. 4C illustrates yet another hierarchical feedback model for aconsumer electronics product. This example illustrates a hierarchicalconfiguration in which various releases of the electronics product arerepresented as nodes 434-436 in a lowest level of the hierarchy with thetype of electronics product being represented as nodes 433, 437 at anext higher level. At a next higher level, the companies that releasethe products are represented as nodes 431, 432 followed by the generalcategory of consumer electronics products.

FIG. 4D illustrates another hierarchical feedback model for gamingapplications 440 in which distribution companies are represented asnodes 441, 442, game titles are represented as nodes 443, 449, releasesof the game title for various years are represented as nodes 444, 445,and versions of a release of the game title are represented as nodes446-448. FIG. 4E illustrates another hierarchical feedback model for acomposite service, in this case an ordering service 450, which iscomprised of component or sub-services represented by nodes 451-453,i.e. an order acceptance service, a seller service, and a paymentservice.

FIGS. 4A-4E illustrate that hierarchical feedback models may beestablished for various offerings including consumer electronicsproducts, software products, composite services, movies, and the like.These are just examples of the types of offerings that may berepresented by hierarchical feedback models in accordance with theillustrative embodiments and other services and products may be likewiserepresented depending upon the particular implementation.

As noted above, the hierarchical feedback model provides the structureby which to determine how aggregate user feedback should be propagatedfrom one node to the next to take into consideration the user feedbackfor components of composite offering (service, product, etc.) thatprovide underlying support for other nodes, i.e. nodes upon which othernodes are dependent. Thus, for example, a user's rating of a paymentservice 453 influences the rating of the composite ordering service 450as a whole. Likewise, a user's rating of the game release FIFA 14 444,available for Electronic Arts Games, influences the user rating for theFIFA game title 443 as a whole, and further influences the user ratingfor the distribution company EA Games 441.

It should be appreciated that there may be times when a node does nothave an associated aggregate user feedback score. For example, a nodethat does not depend on other nodes and which has not itself been ratedby a user may not have an associated user rating. In such cases, themechanisms of the illustrative embodiments may infer an aggregated userfeedback score for the node based on similar nodes found in the same ordifferent registered hierarchical feedback models. Similarity may beevaluated based on a variety of factors including similar names of thenodes, the level relationship between the current node and the similarnode, the hierarchical structure connecting the nodes if any, asimilarity in hierarchical structures associated with the current nodeand the similar node, the nodes with which the similar node is relatedcompared to the nodes with which the current nodes is related, and thelike.

FIG. 5 illustrates example diagrams of hierarchical structures fordifferent emergency response services to illustrate the use ofsimilarity of nodes for inferring feedback information for nodes inaccordance with one illustrative embodiment. In FIG. 5, a hierarchicalfeedback model is established from the Chicago Emergency Response (ER)service 510 which includes sub-services 511 and 512 which correspond toa route planning service and an evacuation service. The route planningservice 511 has sub-services including traffic service 513 and weatherservice 514. The evacuation service 412 has sub-services includingtraffic service 515 and weather service 516. A similar hierarchicalfeedback model is provided for the Phoenix ER service 520, comprisingsimilar nodes 521-526.

Assume that there is no aggregate user feedback available for node 516in FIG. 5, but that there is aggregate user feedback available for nodes514 and 526, as well as the other nodes of the two hierarchical feedbackmodels. In order to generate a more accurate representation of theaggregate user feedback for the evacuation service 512, it is necessaryto take into consideration the contribution from the user feedback forthe weather service 516. However, since there is no user feedbackassociated with the weather service 516, its contribution is notaccurately reflected.

In order to generate an initial aggregate user feedback score for theweather service 516, the mechanisms of the illustrative embodimentssearch for similar nodes in the same or different hierarchical feedbackmodels. The similarity of the nodes are evaluated based on variousfactors with the most similar node being selected and its aggregate userfeedback score being then attributed or associated with the weatherservice node 516. Some factors provide higher levels of similarity thanothers with multiple factors being combined to evaluate a totalsimilarity of the nodes. For example, similarity of node labels may be adominant factor in determining similarity such that nodes with similarlabels, e.g., “weather service” in node 514 and 526 being similar to“weather service” in node 516, are determined to be strongly similar.Moreover hierarchical structure associated with the nodes may beevaluated to determine similarity such that nodes in a same level oftheir respective hierarchical feedback models are determined to be morestrongly similar. Furthermore, sibling and cousin type relationships maybe evaluated, where a “sibling” is a node that is within the sameprimary branch from a root node as the other node and “cousins” arenodes that are in different branches of root node. For example, in FIG.5, node 514 is a cousin to node 516.

In addition if the nodes are in different hierarchical feedback models,but the majority of the structure associated with the nodes in thedifferent hierarchical feedback models is similar, i.e. same nodes withsame node labels are connected in a similar hierarchical arrangement,then the nodes may be considered to be strongly similar. For example, inFIG. 5, due to the strong similarity between the structure in the twohierarchical feedback models associated with nodes 516 and 526, and thesimilarity of the labels of the nodes 516 and 526, it may be determinedthat nodes 516 and 526 are more strongly similar than nodes 516 and 514.This is because, while node 514 has a similar label, node 514 is acousin to node 516 and thus, is associated with a different hierarchicalstructure (nodes 511 and 513) than node 516 (nodes 512 and 515 which aresimilar to nodes 522 and 525). Thus, node 526 may be selected as asimilar node to node 516 and the aggregate user feedback associated withnode 526 may also be attributed to node 516 such that node 516'saggregate user feedback score is set equal to that of node 526.

Of course other similarity evaluations may be utilized without departingfrom the spirit and scope of the illustrative embodiments. Anysimilarity factors may be considered depending upon the implementationas long as it is determined that those factors indicate nodes that aresimilar for purposes of inferring aggregate user feedback scores betweenthe similar nodes.

While FIG. 5 illustrates the two emergency response services as separatehierarchical feedback models, it should be appreciated that theprinciples discussed above and the mechanisms of the illustrativeembodiments are also applicable when these services are part of the sameoverall hierarchical feedback model. For example, a root node, or parentnode of the two models shown in FIG. 5 may be provided, such as “WeatherService” with both the Chicago ER service node 510 and its sub-tree, andPhoenix ER service node 520 and its sub-tree, being child nodes of the“Weather Service” root or parent node in the larger hierarchicalfeedback model. In such a situation, the Chicago ER service 510 andPhoenix ER service 520 may be considered sibling or cousin nodes in thelarger hierarchical model depending upon the particular context.

As discussed above, one of the principle operations of the illustrativeembodiments is to aggregate user feedback information for a node takinginto consideration contributions from other related nodes, such as childnodes, in a hierarchical feedback model. “Related” in this contextrefers to nodes that are connected to the current node either directlyor indirectly along a branch from the root node of the hierarchicalfeedback model. Thus, nodes in other branches that are not connected byway of an intermediate node located within the hierarchical feedbackmodel at a lower level than the root node are not considered “related”.The degree of “relatedness” may be based on a relative distance betweenthe nodes, e.g., number of edges and nodes between the nodes.

In addition to generating aggregate user feedback information for anode, the principle operations of the illustrative embodiments furtherprovide mechanisms for propagating the aggregate user feedback from nodeto node within a hierarchical feedback model based on the weights ofedges between nodes which indicate a degree of influence or dependencythat one node has on another. The propagation is performed from lowerlevels to higher levels (child to parent, or leaf nodes to root node)within the hierarchical feedback model. It should be appreciated thatthere may be many different formulations for how exactly to propagatethe aggregate user feedback of a node in the hierarchical feedback modelto other nodes in the hierarchical feedback model, any of which areintended to be within the spirit and scope of the illustrativeembodiments. FIGS. 6A-6C are provided hereafter to illustrate onepossible implementation for performing such aggregate user feedbackgeneration and propagation in accordance with one illustrativeembodiment.

As stated, FIGS. 6A-6C are example diagrams of a generalizedhierarchical feedback model for illustrating aggregate feedbackgeneration and propagation in accordance with one illustrativeembodiment. In the example hierarchical feedback model of FIGS. 6A-6C,an offering X (e.g., a service, product, or the like) has componentofferings Y1 and Y2. Each of the component offerings Y1 and Y2 havetheir corresponding sub-component offerings Z1, Z2, Z3, and Z4. Thus, ahierarchy for propagating feedback is established in which the offeringX is at the highest level (or root level), Y1 and Y2 are at anintermediate level, and Z1-Z4 are at a lowest level of the hierarchy. Itshould be appreciated that this example hierarchy is kept simple byhaving a single intermediate level for illustration purposes and actualhierarchies may comprise many intermediate levels with the resultinghierarchical feedback model being very complex. In addition, forsimplicity of the following description, the weights of each of theedges is assumed to be 1, however in actual implementation one or moreof the edges may have differing weights less than or greater than 1,depending upon the implementation, which may be applied to the childcontributions of aggregate user feedback as weighting factors whenperforming propagation of user feedback from node to node.

Before beginning a more detailed discussion of the user feedbackaggregation and propagation calculations performed in this illustrativeembodiment, it is useful to first have an understanding of the variousvalues that will be used in the calculations. The following is a listingof the values and their descriptions as referenced in the equations andcalculations described hereafter:

-   S(x) is the user feedback score for node x;-   B(x) is the Bayesian estimation of node x;-   C(x) is the aggregate user feedback score contribution of node x's    child nodes;-   R(x) is the average rating of node x;-   U(x) is the average rating of the universe of node x, where the    “universe” comprises the nodes at a same level of the hierarchical    feedback model as the node x;-   Ra_(x) is the rating value for node x;-   Re_(x) is the relevance value for node x;-   V(x) is the cumulative number of votes for node x;-   V_(x) is the total number of votes for node x, where “votes”    represent the weighted sum of the number of users that provide    feedback for node x;-   C_(x) is a child node of node x; and-   d(x, y) is the number of edges between node x and node y.

With regard to FIG. 6A, assume that the sub-component offering Z1receives a set of user feedback information from 4 users, via thefrontend engine of the illustrative embodiments, that comprises thetuples (rating, relevance) of [(4,5), (4,3), (5,4), (3,3)] as shown.Thus, each user feedback has a value for the user's subjectiveperception of quality (rating) and usefulness to the user's needs(relevance) on a defined scale which, in this example, is a 5-star scalesuch that each value may have an integer setting from 0 to 5. Theaggregate user feedback score for sub-component offering Z1 may becalculated using an equation such as that shown as equation 1 belowwhich combines Z1's contribution B(Z1) as a Bayesian estimation, withZ1's child node contributions C(Z1):

$\begin{matrix}{{{S\left( z_{1} \right)} = {{a \times {B\left( z_{1} \right)}} + {\left( {1 - \alpha} \right) \times {C\left( z_{1} \right)}}}},\left\{ \begin{matrix}{a = \frac{3}{4}} & {{{if}\mspace{14mu} z_{1\;{children}}} \neq \phi} \\{a = 1} & {{{if}\mspace{14mu} z_{1\;{children}}} = \phi}\end{matrix} \right.} & \left( {{Eq}.\mspace{14mu} 1} \right)\end{matrix}$The value for α is a parameter that assigns weight to the contributionsdepending on the particular implementation. In the above example, thisvalue is set to 1 if Z1 has no children and thus, the aggregate userfeedback score for Z1 is entirely based on its own direct user feedback.If Z1 has children, then α is set to ¾ so that the aggregate feedbackfor node Z1 is primarily based on the user feedback directly for theoffering corresponding to node Z1 but is also based partly (¼) on theuser feedback associated with the child nodes of node Z1. It should benoted that the particular values for α discussed above are just examplesand are not intended to state or imply any limitation with regard to thepossible weighting factors that may be utilized, i.e. other values maybe used without departing from the spirit and scope of the illustrativeembodiments. For example, in other illustrative embodiments, the valuefor α may be set to any value in the range of 0.5 to 1.0 when Z1 haschild nodes, depending upon the particular implementation. In somecases, the value of α may be set based on the number of child nodes thatZ1 has such that the value for α is reduced as the number of child nodesincreases. Other ways of determining what value to give to a may be usedwithout departing from the spirit and scope of the illustrativeembodiments.

The value for node Z1's direct aggregate user feedback (user feedbackspecifically directed to node Z1 as opposed to node Z1's childofferings), i.e. B(Z1), in one illustrative embodiment, is aggregatedusing a Bayesian estimation model for aggregation as shown in Equation 2below:

$\begin{matrix}{{{B\left( z_{1} \right)} = {{\beta \times {R\left( z_{1} \right)}} + {\left( {1 - \beta} \right) \times {U\left( z_{1} \right)}}}},\left\{ \begin{matrix}{{\beta--} > 0} & {{{if}\mspace{14mu}{v--}} > 0} \\{{\beta--} > 1} & {{{if}\mspace{14mu}{v--}} > \infty}\end{matrix} \right.} & \left( {{Eq}.\mspace{14mu} 2} \right)\end{matrix}$The Bayesian estimation was selected for the illustrative embodimentbecause if there is not enough direct user feedback for the node Z1,then the feedback of nodes which are similar to Z1 may be used to get amore accurate user feedback score. Due to similarity, there is anexpectation that Z1's feedback and score will be similar to these nodes.That is, node Z1's user feedback score is aggregated based on itsaverage user feedback and the average user feedback of Z1's universeU(Z1). The value β is a parameter that assigns weight to the averagesdepending on the particular implementation desired. For example, β tendsto 0 if there are not enough votes v (number of users providingfeedback) and thus, most of the weight is given to the universe averageU(Z1). On the other hand, the value β tends to 1 if there are enoughvotes, and most of the weight is then given to Z1's own averagefeedback. It should be appreciated that the value for β may thus, bedynamically determined to be a particular value in the range from 0 to 1based on the number of votes a particular node has at the time of thecalculation.

The average rating user feedback for node Z1, i.e. R(Z1), in oneillustrative embodiment, is calculated using a relationship such asshown in Equation 3. Equation 3 provides a weighted average of Z1'sratings (Ra_(i)), where the weight of each rating comes from therespective relevance (Re_(i)):

$\begin{matrix}{{R\left( z_{1} \right)} = \frac{\sum\limits_{i = 1}^{k}\;{{Ra}_{i} \times {Re}_{i}}}{\sum\limits_{i = 1}^{k}\;{Re}_{i}}} & \left( {{Eq}.\mspace{14mu} 3} \right)\end{matrix}$

Thus, taking the example user feedback tuples for node Z1 mentionedabove, each rating feedback is multiplied by its corresponding relevancefeedback and the result is summed with the products of the other userfeedback tuples. The sum is then divided by the sum of the relevanceacross all of the tuples:R(Z1)=((4*5)+(4*3)+(5*4)+(3*3))/(5+3+4+3)=4.1Thus, the aggregate of the direct user feedback for node Z1 is equal to4.1 in this example.

In order to calculate node Z1's universe average feedback U(Z1), theuniverse of node Z1 is first defined as the set of nodes which havecertain similarity with node Z1. For instances, in FIG. 6A, nodes Z1 andZ2 are sibling nodes and thus, have some similarity, e.g., multipleversions of the same song (regular, remix, and unplugged). Further, itis possible for node Z1 to have similarity with cousin nodes Z3 and Z4,e.g., sports gaming applications by the same vendor (FIFA and NHL byElectronic Arts Sports, for example). The universe of a node is aparameter that may be defined in any way suitable to the particularimplementation such that the universe comprises nodes that have adesired degree of similarity. In the depicted example, the universe of anode is defined as the sibling and cousin nodes of the node in question,where sibling nodes are nodes that have edges to a same parent node asthe node in question and are at a same level of the hierarchicalfeedback model as the node in question, and cousin nodes are nodes atthe same level of the hierarchical feedback model as the node inquestion, but have edges to nodes that do not have a direct edge to acommon parent node as the node in question.

A similarity weight is assigned to the nodes in the universe based onthe particular implementation and definition of the universe. Forexample, the similarity weight for node Z1's cousin nodes is set to alower weight than sibling nodes because they are more distant from nodeZ1 in the hierarchical feedback model. Thus, for example, assuming thatnode Z1's universe contains all of its siblings and cousin nodes, withsibling nodes having equal similarity weight and cousin nodes havinghalf similarity weight (this is just an example and the similarityweights can be adjusted for different implementations), and each siblinghaving k user feedback and each cousin having p user feedback, theuniverse average feedback for node Z1, i.e. U(Z1), may be calculatedusing Equation 4 as follows:

$\begin{matrix}{{U\left( z_{1} \right)} = \frac{{\sum\limits_{i = 1}^{m}\;\left( {\sum\limits_{j = 1}^{k}\;{{Ra}_{j} \times {Re}_{j}}} \right)_{i}} + {\frac{1}{2}{\sum\limits_{i = 1}^{n}\;\left( {\sum\limits_{j = 1}^{p}\;{{Ra}_{j} \times {Re}_{j}}} \right)_{i}}}}{{\sum\limits_{i = 1}^{m}\;\left( {\sum\limits_{j = 1}^{k}\;{Re}_{j}} \right)_{i}} + {\frac{1}{2}{\sum\limits_{i = 1}^{n}\;\left( {\sum\limits_{j = 1}^{p}\;{Re}_{j}} \right)_{i}}}}} & \left( {{Eq}.\mspace{14mu} 4} \right)\end{matrix}$Thus, the R(Z1) and U(Z1) values may be utilized in Equation 2 above toobtain the Bayesian estimation of the aggregate user feedback score fornode Z1. This value may then be utilized to perform the calculationaccording to Equation 1 above which, because node Z1 has no child nodes,results in the aggregate user feedback score S(Z1) being equal to B(Z1).Thus, utilizing the above equations, for purposes of continuing thepresent example, it is assumed that U(Z1)=4, β=0.4 and α=1, R(Z1)=4.1,B(Z1)=0.4*4.1+(1−0.4)*4=4, and S(Z1)=1*4+(1−1)*C(Z1)=4. A similarcalculation can be made for the aggregate user feedback scores for nodesZ2, Z3, and Z4, i.e. S(Z2), S(Z3), and S(Z4). For purposes of thepresent example, it will be assumed that similar calculations are madefor each of the nodes Z2, Z3, and Z4 which result in aggregate userfeedback scores of S(Z2)=3.5, S(Z3)=4.5, and S(Z4)=3, as shown in FIG.6B.

The aggregated user feedback scores of nodes are propagated up all theway to the root and diminish with each level of the hierarchicalfeedback model. For example, with regard to node Y1, the aggregated userfeedback score calculated by applying Equation 1 combines node Y1'scontribution and contributions from node Y1's child nodes Z1 and Z2.Node Y1's contribution, from its direct user feedback, can be calculatedby applying Equation 2 above. The contribution from node Y1's childnodes C(Y1), in one illustrative embodiment, is calculated usingEquation 5 hereafter which uses a weighted average to aggregate theaggregated user feedback scores of the child nodes. Each child node'sscore's weight is decided based on the amount of user feedback that theparticular child node has received, i.e. the number of votes v(c_(i))associated with that child node:

$\begin{matrix}{{C\left( y_{1} \right)} = \frac{\sum\limits_{i = 1}^{k}\;{{S\left( c_{i} \right)} \times {V\left( c_{i} \right)}}}{\sum\limits_{i = 1}^{k}\;{V\left( c_{i} \right)}}} & \left( {{Eq}.\mspace{14mu} 5} \right)\end{matrix}$

The number of cumulative votes V(Y1) for node Y1 are calculated usingEquation 6 hereafter which is based on node Y1's own votes and thecumulative votes of node Y1's child nodes:

$\begin{matrix}{{{V\left( y_{1} \right)} = {V_{y\; 1} + {\sum\limits_{i = 1}^{k}\;\frac{V\left( c_{i} \right)}{2^{d}\left( {c_{i},y_{1}} \right)}}}},\left\{ {{d\left( {c_{i},y_{1}} \right)} = {e}_{c_{i}}^{y_{1}}} \right\}} & \left( {{Eq}.\mspace{14mu} 6} \right)\end{matrix}$The relative contribution of a child node's cumulative votes diminishesas the votes are propagated to higher levels, based on their distancefrom the node being evaluated. The distance d(C_(i), Y_(i)) iscalculated as the number of edges between the nodes C_(i) and Y_(i). Forinstances, only ¼ of node Z2's cumulative votes contribute to node X'scumulative votes. Node Y1's own votes are calculated using Equation 7 asthe sum of all the relevance user feedback input (Re_(i)) node Y1receives:

$\begin{matrix}{V_{y_{1}} = {\sum\limits_{i = 1}^{k}\;{Re}_{i}}} & \left( {{Eq}.\mspace{14mu} 7} \right)\end{matrix}$

Thus, with reference again to FIG. 6B, it is assumed that node Z1 has300 votes, and node Z2 has 200 votes, i.e. V(Z1)=300 and V(Z2)=200. IfB(Y1)=3, S(Z1)=4, and S(Z2)=3.5, then C(Y1) using Equation 5 above iscalculated as C(Y1)=((4*300)+(3.5*200))/(300+200)=3.8. Now, if α=0.6,the aggregate user feedback score for node Y1 is calculated asS(Y1)=0.6*3+(1−0.6)*3.8=3.3. A similar calculation can be performed withregard to node Y2 with nodes Z3 and Z4 having 20 and 30 votes,respectively, and scores of 4.5 and 3, respectively, to thereby generatean aggregate user feedback score for node Y2 as S(Y2)=3.9, as shown inFIG. 6B.

With reference now to FIG. 6C, assume that node Y1 has 100 votesassociated directly with node Y1, i.e. V_(y1)=100 (see Equation 7above). Using Equation 6 above to calculate the cumulative votes fornode Y1, one obtains V(Y1)=100+(300/2¹)+(200/2¹)=350. Thus, thecumulative votes for node Y1 is 350. A similar calculation may be madefor node Y2, assuming that V_(y2)=40, such that the cumulative votes fornode Y2 is 65 votes, i.e. V(Y2)=40+(20/2¹)+(30/2¹)=65.

Continuing the propagation up to node X, using Equations 1-7 above, oneobtains the following values:C(X)=((3.3*350)+(3.9*65))/(350+65)=3.4Let B(X)=4.5 and V_(X)=25V(X)=25+350/2+65/2=233 votesLet α=0.5S(X)=0.5*4.5+(1−0.5)*3.4=4

Thus, through the mechanisms of the illustrative embodiments, algorithmsare provided for calculating the aggregate user feedback for a componentof a composite offering, such as a service or product, based on thedirect user feedback for that component and contributions fromsub-components related to that component. Moreover, similar componentsmay also be taken into consideration in cases where the amount of userfeedback for a component is not sufficiently high enough, e.g., siblingand cousin components. All of these factors combine to generate anaggregate user feedback score for the particular component. Thisaggregate user feedback score for the particular component may furtherbe propagated up a hierarchical feedback model so as to propagate theinfluence of the components' aggregate user feedback to other componentsthat depend on it. This provides a much more accurate model of theactual user feedback for components taking into consideration the userfeedback of the components upon which they depend in a hierarchicalmanner.

FIG. 7 is a flowchart outlining an example operation for providingon-demand hierarchical feedback aggregation in accordance with oneillustrative embodiment. The operation outlined in FIG. 7 may beperformed using a hierarchical feedback aggregation system such as thatshown in FIG. 3 above with operations being distributed between thefront end engine and the backend engine in the manner previouslydescribed. As such, the operation outlined in FIG. 7 may be implementedusing software executed on hardware of one or more data processingsystems or computing devices, dedicated hardware of one or more dataprocessing systems or computing devices, or any combination of softwareand hardware elements of data processing systems or computing devices.

It should be appreciated that FIG. 7 represents a single iteration ofthe operation in accordance with one illustrative embodiment. Theoperation outlined in FIG. 7 may be repeated many times based on theneeds of the provider and the operation of the composite offering. Forexample, steps 740-830 may be repeated many times based on a periodicexecution of these operations, in response to each new user feedbackreceived, or the like.

As shown in FIG. 7, the operation starts with a composite offeringhierarchical feedback model being defined (step 710). As noted above,this definition may be performed using a graphical user interface tooloffered via an on-demand registration/request engine 312 of a backendengine 310 of the HFA system, for example, through which a user mayconstruct the hierarchical model as a set of nodes and edges connectingnodes, with initial weights assigned to the nodes based on an initialestimation of the level of dependence that one node has on another withregard to user feedback.

The model is registered with the backend engine for use in collecting,calculating, aggregating, and propagating user feedback (step 720). Thefrontend engine is then configured to collect user feedback and presentuser feedback to users for the particular composite offering (step 730).This operation may involve customizing general templates utilized by thefrontend engine to present questions to users, deploying the frontendengine into a software environment of the composite offering or systemthat provides the composite offering, as well as any other configurationoperations that enable the frontend engine to operate within theproviders computing system(s) to collect user feedback information, andoptionally user profile information, and provide this information to thebackend engine.

User (or consumer) feedback for a component in the composite offering isreceived (step 740) and is checked for validity (step 750), e.g., bychecking the user's demographics against the desired demographicsspecified by the offering provider, by comparing the user's credibilitywith a threshold level of credibility by evaluating the user's historyof user feedback, or any other criteria that may used to accept orreject the user's feedback as being feedback upon which the mechanismsof the illustrative embodiments may be implemented. If the user'sfeedback is not valid, the feedback is discarded (step 760) and theoperation ends or, alternatively, may return to step 740 and continuemonitoring for further user feedback from other users.

If the user feedback is valid, then the feedback is submitted to thebackend engine (step 770) which then aggregates the user feedback forthe component based on contributions from user feedback for thatcomponent and its sub-components (step 780). The resulting aggregateuser feedback for the component is then propagated to the parentcomponents based on the hierarchical feedback model (step 790). As notedby the dashed lines in FIG. 7, this may cause a recursive operationwhere the parent nodes' aggregate user feedback is then updated andpropagated by way of repeated executions of steps 780-790. Moreover, theupdated user feedback for the component, and potentially the parentcomponents, may be used to update a display of the aggregated userfeedback for components on the frontend engine (step 830).

At step 800, a query may be received to access the aggregated userfeedback from the backend engine. This query may be a query from a uservia the frontend engine, a query from a provider of the compositeoffering via an administrator console, a query from an administrator ofthe HFA system, or the like. The query may request certain analyticaloperations to be performed, such as providing an average of the userfeedback across all components of the composite offering, a particulargraphical display of the user feedback, a filtered display of the userfeedback, e.g., only user feedback that is less than a specifiedthreshold, or the like. The requested analytics are performed and anoutput to support decision making is generated and output (step 810). Inaddition, the composite offering registration node and edge weights maybe updated based on the results of the analytics (step 820). Theoperation then terminates.

Thus, the illustrative embodiments provide mechanisms for providingon-demand user feedback aggregation and propagation in a compositeoffering. The mechanisms of the illustrative embodiments utilizemultiple different factors to represent the user feedback, such asquality rating and relevance rating. The mechanisms of the illustrativeembodiments further utilize a hierarchical feedback model of thecomposite offering to facilitate propagation of user feedback tocomponents of the composite offering based on specified dependencies andrelationships between the components. Thus, a more accuraterepresentation of the actual user experience with the various componentsof the composite offering is achieved.

It should be appreciated that while examples of offerings have beenprovided in the description and drawings, the illustrative embodimentsare not limited to these specific examples and any offering may be thesubject of the mechanisms of the illustrative embodiments. For example,various hierarchical feedback models may be generated and used with themechanisms of the illustrative embodiments for various types ofofferings including, but not limited to, services, applications,artistic works (e.g., movies, books, musical compositions, etc.),electronic products, business establishments (e.g., restaurants, retailstores, corporations), information technology solution packages, or anyother type of offering.

As noted above, it should be appreciated that the illustrativeembodiments may take the form of an entirely hardware embodiment, anentirely software embodiment or an embodiment containing both hardwareand software elements. In one example embodiment, the mechanisms of theillustrative embodiments are implemented in software or program code,which includes but is not limited to firmware, resident software,microcode, etc.

A data processing system suitable for storing and/or executing programcode will include at least one processor coupled directly or indirectlyto memory elements through a system bus. The memory elements can includelocal memory employed during actual execution of the program code, bulkstorage, and cache memories which provide temporary storage of at leastsome program code in order to reduce the number of times code must beretrieved from bulk storage during execution.

Input/output or I/O devices (including but not limited to keyboards,displays, pointing devices, etc.) can be coupled to the system eitherdirectly or through intervening I/O controllers. Network adapters mayalso be coupled to the system to enable the data processing system tobecome coupled to other data processing systems or remote printers orstorage devices through intervening private or public networks. Modems,cable modems and Ethernet cards are just a few of the currentlyavailable types of network adapters.

The description of the present invention has been presented for purposesof illustration and description, and is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the describedembodiments. The embodiment was chosen and described in order to bestexplain the principles of the invention, the practical application, andto enable others of ordinary skill in the art to understand theinvention for various embodiments with various modifications as aresuited to the particular use contemplated. The terminology used hereinwas chosen to best explain the principles of the embodiments, thepractical application or technical improvement over technologies foundin the marketplace, or to enable others of ordinary skill in the art tounderstand the embodiments disclosed herein.

What is claimed is:
 1. A method, in a hierarchical feedback aggregation(HFA) system implemented in a plurality of data processing systems, eachdata processing system comprising a processor and a memory, forcollecting and presenting user feedback information for a computerprovided composite offering, the method comprising: registering, by abackend engine of the HFA system executing in a first data processingsystem, a hierarchical feedback model for the computer providedcomposite offering, wherein the computer provided composite offering isone of a composite service or composite product comprising a pluralityof computer implemented components operating in one or more computingdevices, wherein the plurality of computer implemented componentscomprise at least one of computer implemented sub-services or computerimplemented sub-products, and wherein the hierarchical feedback modelcomprises nodes representing the computer implemented components of thecomputer provided composite offering and edges representingrelationships between the computer implemented components; receiving,via a frontend engine of the HFA system executing in a second dataprocessing system providing at least one computer implemented componentof the computer provided composite offering, user feedback for anidentified computer implemented component of the computer providedcomposite offering; generating, by the backend engine of the HFA system,an aggregate user feedback score for the identified computer implementedcomponent based on a combination of the user feedback for the identifiedcomputer implemented component and aggregate user feedback scores forchild components of the identified computer implemented component in thehierarchical feedback model; dynamically updating, by the backend engineof the HFA system, the hierarchical feedback model, to generate amodified hierarchical feedback model, based on the aggregate userfeedback score for the identified computer implemented component,wherein dynamically updating the hierarchical feedback model comprisesupdating weights associated with edges in the hierarchical feedbackmodel based on a function of a usage metric associated with theidentified computer implemented component, and a distance of the edgefrom a first node, in the hierarchical feedback model, corresponding tothe identified computer implemented component; and outputting, by thebackend engine, a representation of the generated aggregate userfeedback score for the identified computer implemented component fordecision support utilization, wherein the representation furthercomprises a representation of the modified hierarchical feedback model.2. The method of claim 1, further comprising: propagating, by thebackend engine of the HFA system, the aggregate user feedback score forthe identified computer implemented component up the hierarchicalfeedback model to one or more parent nodes of the hierarchical feedbackmodel from which the first node corresponding to the identified computerimplemented component is dependent.
 3. The method of claim 2, whereinpropagating the aggregate user feedback score for the identifiedcomputer implemented component up the hierarchical feedback modelcomprises, for a parent node in the one or more parent nodes,calculating an aggregate user feedback score for the parent node basedon a combination of the aggregate user feedback score for the identifiedcomputer implemented component and a user feedback score associated withthe parent node.
 4. The method of claim 1, wherein the user feedback forthe identified computer implemented component comprises a first userfeedback part corresponding to a quality rating and a second userfeedback part corresponding to a relevance rating, and whereingenerating the aggregate user feedback score for the identified computerimplemented component comprises generating the aggregate user feedbackscore as a function of the quality rating weighted by the relevancerating.
 5. The method of claim 4, wherein generating the aggregate userfeedback score further comprises weighting the user feedback for theidentified computer implemented component based on an evaluation ofcredibility of a user that provided the user feedback.
 6. The method ofclaim 1, wherein the hierarchical feedback model is a first hierarchicalfeedback model, and wherein at least one child component of theidentified computer implemented component does not have an aggregateuser feedback score provided based on direct user feedback provided forthe at least one child component, and wherein for the at least one childcomponent, an aggregate user feedback score is inferred by identifyingat least one similar component in a second hierarchical feedback model,associated with another computer provided composite offering, that issimilar to the first hierarchical feedback model, and associating anaggregate user feedback score of the at least one similar component withthe at least one child component.
 7. The method of claim 6, whereinidentifying at least one similar component comprises identifying atleast one of a similar hierarchical structure within the similarhierarchical feedback model, or a similar name associated with a secondnode within the similar hierarchical feedback model.
 8. The method ofclaim 1, wherein the user feedback for the identified computerimplemented component is received by the frontend engine of the HFAsystem from a component specific console implemented in the second dataprocessing system, and wherein the component specific console interfaceswith a user to collect the user feedback from the user via presentationof one or more questions requesting the user feedback, and provides thecollected user feedback to the frontend engine.
 9. The method of claim1, wherein the user feedback for the identified computer implementedcomponent is received directly by the frontend engine by using one ormore templates to present questions to a user to collect the userfeedback, and wherein the one or more templates are customized for theidentified component.
 10. The method of claim 5, further comprisingreceiving, from a provider of the computer provided composite offering,one or more identifiers of one or more characteristics of a target userof the computer provided composite offering, wherein weighting the userfeedback for the identified component based on the evaluation ofcredibility of the user that provided the user feedback furthercomprises weighting the user feedback based on a correspondence of oneor more characteristics of the user that provided the user feedback withthe one or more characteristics of the target user.
 11. The method ofclaim 1, wherein the aggregate user feedback score is generated based ona function of direct user feedback and user feedback for similarcomponents of the computer provided composite offering, wherein similarcomponents are computer implemented components associated with similarnodes that are either cousin nodes at a same level of the hierarchicalfeedback model as an identified node corresponding to the identifiedcomputer implemented component, but do not have direct edges to a sameparent node as the identified node, or sibling nodes that are nodes inthe hierarchical feedback model that have edges connecting them to thesame parent node as the identified node.
 12. The method of claim 11,wherein similar nodes are weighted by a similarity weight in thefunction, and wherein the similarity weight is based on a type ofsimilarity.
 13. The method of claim 12, wherein sibling nodes have afirst similarity weight different from cousin nodes which have a secondsimilarity weight.
 14. The method of claim 13, wherein the firstsimilarity weight is higher than the second similarity weight.
 15. Themethod of claim 2, wherein the aggregate user feedback score ispropagated up to a root node of the hierarchical feedback model and acontribution of the aggregate user feedback to user feedback for higherlevel nodes is diminished with each higher level of the hierarchicalfeedback model.
 16. The method of claim 1, wherein receiving userfeedback for the identified computer implemented component of thecomputer provided composite offering comprises receiving different userfeedback of different feedback types, and wherein generating theaggregate user feedback score comprises utilizing a correlation functionthat correlates feedback types having different scales of feedback to acommon aggregate user feedback score scale.
 17. The method of claim 15,wherein, for each higher level node, the contribution of the aggregateuser feedback to the user feedback for the higher level node isdiminished by an amount dependent upon a distance of the higher levelnode from a third node corresponding to the aggregate user feedback inthe hierarchical feedback model.
 18. The method of claim 1, wherein theplurality of computer implemented components comprise a plurality ofcomputer implemented software services executing on different computingdevices, and wherein at least one of the computer implemented softwareservices provides input data to one or more other computer implementedsoftware services in the plurality of computer implemented softwareservices.
 19. The method of claim 1, wherein the plurality of computerimplemented components comprise at least two different computerimplemented components provided by at least two different computingdevices of the one or more computing devices.
 20. The method of claim17, wherein receiving user feedback for the identified computerimplemented component of the computer provided composite offeringcomprises automatically modifying one or more general templates, forpresenting questions to a user to solicit feedback, to be customized forthe identified computer implemented component.